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Abstract. The two major approaches to sparse recovery are Li-minimization and greedy methods. 
Recently, NeedeU and Vershynin developed Regularized Orthogonal Matching Pursuit (ROMP) 
that has bridged the gap between these two approaches. ROMP is the first stable greedy algorithm 
providing uniform guarantees. 

Even more recently, NeedeU and Tropp developed the stable greedy algorithm Compressive 
Sampling Matching Pursuit (CoSaMP). CoSaMP provides uniform guarantees and improves upon 
the stability bounds and RIC requirements of ROMP. CoSaMP offers rigorous bounds on compu- 
tational cost and storage. In many cases, the running time is just 0(A'' log A*'), where A'' is the 
ambient dimension of the signal. This review summarizes these major advances. 



1. Introduction 

Sparse signals are those that contain much less information than their ambient dimension sug- 
gests. The conventional signal compression scheme acquires the entire signal and then compresses 
it. This methodology has been questioned for decades, and new approaches in compressed sensing 
have been developed to overcome this seemingly wasteful approach. 

Suppose a; is a signal in M^, and define the £o quasi- norm^ 

lla^llo = |supp(a;)| = \{j : xj / 0}| . 

When ||a;||Q < s, we say that the signal x is s-sparse. In practice, signals are not exactly sparse, but 
are rather close to sparse vectors. For example, compressible signals are those whose coefficients 
decay rapidly when sorted by magnitude. We say that a signal x is p- compressible with magnitude 
R if the sorted components of the signal decay at the rate 

< -R-r^/f for i = 1,2,3, ... . (1.1) 

The sparse recovery problem is the reconstruction of such signals from a set of nonadaptive 
linear measurements. The measurements are of the form ^x where ^ is some mx N measurement 
matrix. Although in general this recovery is NP-Hard, work in compressed sensing has shown that 
for certain kinds of measurement matrices, recovery is possible when the number of measurements 
m is nearly linear in the sparsity s, 

m = slog'^^^\N). (1.2) 

The survey [2] contains a discussion of these results. 

The two major algorithmic approaches to sparse recovery are based on Li-minimization and on 
greedy methods (Matching Pursuits). In this review we briefly describe these methods, as well 
as two new iterative methods that provide the advantages of both approaches. The first method, 
ROMP, is the first stable algorithm to provide uniform guarantees. The second method, CoSaMP, 
improves upon the results of ROMP, and provides rigorous computational bounds. 
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2. Major approaches 

2.1. Li-minimization. To recover the sparse signal x from its measurements ^x, one needs to 
find the the solution to the highly non-convex problem 

min||z||o subject to <^z = (^x. (Lq) 

Donoho and his associates [6] suggested that for some measurement matrices the generally 
NP-Hard problem (Lq) should be equivalent to its convex relaxation: 

min||z||i subject to <^z = (^x, (Li) 

where ||2:||i = \zi\ denotes the ^i-norm. The convex problem (Li) can be solved using methods 
of linear programming. 

Clearly if the measurement matrix $ is one-to-one on all 2s-sparse vectors, then the s-sparse 
signal X will be recovered by solving {Lq). Candes and Tao [4] proved that if $ satisfies a stronger 
condition then recovery is possible by solving the convex problem (Li). 

Definition 2.1 (Restricted Isometry Condition). A measurement matrix $ satisfies the Restricted 
Isometry Condition (RIC) with parameters {n,5) for 5 £ (0, 1) if we have 

(1 — (5)||i>||2 < ll^'i'lb < (1 + '^)||'i'||2 for all n-sparse vectors. 

When 6 is small, the restricted isometry condition says that every set of n columns of $ is 
approximately an orthonormal system. It has been shown (see [13] and [7]) that random Gaussian, 
Bernoulli and partial Fourier matrices satisfy the restricted isometry condition with number of 
measurements as in (1.2). In the more practical case when x is not exactly sparse and corrupted 
with noise, we consider the mathematical program: 

min||y||j^ subject to — u||2<e. (2-1) 

This program approximately recovers the signal x even in the presence of noise. 

Theorem 2.2 (Recovery under RIC [3]). Assume that the measurement matrix $ satisfies the 
Restricted Isometry Condition with parameters (3s, 0.2). Let ^ be a measurement matrix and let 
u = ^x + e be a noisy measurement vector where x G is an arbitrary signal and ||e||2 < £■ 
Then the program (2.1) produces an approximation a;* that satisfies: 

1 

where Xg denotes the s-sparse vector consisting of the s largest coefficients in magnitude of x. 

In [5], Candes sharpened this theorem to work under the restricted isometry condition with 
parameters (2s, \/2— 1)- This theorem also demonstrates that in the noiseless case, the Li approach 
provides exact reconstruction. This was proved initially in [4]. 

2.2. Orthogonal Matching Pursuit (OMP). An alternative approach to sparse recovery is via 
greedy algorithms. These methods find the support of the signal x iteratively, and reconstruct the 
signal using the pseudoinverse. 

Orthogonal Matching Pursuit (OMP) is such an algorithm, analyzed by Gilbert and Tropp in [14]. 
Since we expect the columns of the measurement matrix $ to be approximately orthonormal, ^*^x 
is locally a good approximation to x. OMP uses this idea to compute the support of a s-sparse 
signal X. First, the residual r is set to the measurement vector u. At each iteration, the observation 
vector is set, y = <I>*r, and the coordinate of its largest coefficient in magnitude is added to the 
index set /. Then by solving a least squares problem, the residual is updated to remove this 
coordinate's contribution, 

y = arg min ||u — $^||2; r = u — ^y. 



X — a;* 


< C 
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Repeating this s times yields an index set of s coordinates corresponding to the support of the 
signal X. Tropp and Gilbert [14] showed that OMP recovers a sparse signal with high probability. 

Theorem 2.3 (OMP Recovery [14]). Let $ be a m x N subgaussian matrix, and fix a s-sparse 
signal x e . Then OMP recovers (the support of) x from the measurements u = ^x correctly 
with high probability, provided the number of measurements is m ^ slogN . 

2.3. Advantages and challenges of both approaches. The Li -minimization method provides 
uniform guarantees for sparse recovery. Once the measurement matrix <I> satisfies the restricted 
isometry condition, this method works correctly for all sparse signals x. The method is also stable^ 
so it works for non-sparse signals such as those which are compressible, as well as noisy signals. 
However, the method is based on linear programming, and there is no strongly polynomial time 
algorithm in linear programming yet. 

OMP on the other hand, is quite fast both provably and empirically. The speed of OMP is a 
great advantage, but it lacks the strong guarantees that Li provides. Indeed, OMP works correctly 
for a fixed signal and measurement matrix with high probability, and so it must fail for some sparse 
signals and matrices [12]. It is also unknown whether OMP succeeds for compressible signals or on 
noisy measurements. 

There has thus existed a gap between the approaches. The development of Regularized Orthog- 
onal Matching Pursuit (ROMP) bridges this gap by providing a greedy algorithm with the same 
advantages as the Li method. Compressive Sampling Matching Pursuit (CoSaMP) improves upon 
these results and provides rigorous runtime guarantees. We now discuss these new algorithms. 

3. Regularized OMP 

Regularized Orthogonal Matching Pursuit (ROMP) is a new algorithm developed by Needell and 
Vershynin in [11, 10] for sparse recovery that performs correctly for all measurement matrices $ 
satisfying the restricted isometry condition, and for all sparse signals. Again, since the restricted 
isometry condition gaurantees every set of s columns forms approximately an orthonormal system, 
every s coordinates of the observation vector y = are in a loose sense good estimators of 

the corresponding s coefficients of x. This notion suggests to use the s largest coefficients of the 
observation vector y rather than only the largest, as in OMP. We also include a regularization 
step to ensure that each coordinate carries close to an even share of the information. The ROMP 
algorithm is described in Algorithm 3.1. 

For measurement matrices that satisfy the RIC, ROMP stably recovers all s-sparse signals. This 
is summarized in the following theorem from [10]. 

Theorem 3.1 (Recovery by ROMP [10]). Assume a measurement matrix $ satisfies the restricted 
isometry condition with parameters (8s, e) for e = O.Ol/Vlog s. Consider an arbitrary vector x in 
M^. Suppose that the measurement vector ^x becomes corrupted, so we consider u = ^x + e where 
e is some error vector. Then ROMP produces a good approximation x to x: 

\\x - x\\2 < Cy^(\\e\\2 + ''"^ ^'''^ )- (3-1) 

In the case where the signal x is exactly sparse without noise, this theorem guarantees exact 
reconstruction. Note also that in the noisy case, ROMP needs no knowledge about the error vector 
e to approximate the signal. In the special case where a; is a compressible signal as in (1.1), the 
theorem provides the bound 

\\x-x\\2<R'^^^ + Cy^s\\e\\2. (3.2) 

ROMP thus provides the first greedy approach with these uniform and stable guarantees. See [11, 
10] for runtime analysis and empirical results. The results are optimal up to the logarithmic factors. 
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Algorithm 3.1: ROMP Recovery Algorithm [11] 



ROMP($, u, s) 

Input: Measurement matrix measurement vector u, sparsity level s 
Output: Index set / C {1, . . . ,d} 



Initialize: Let the index set / = and the residual r = u. 

Repeat the following steps s times or until |/| > 2s: 
Identify: Choose a set J of the s biggest coordinates in magnitude of the observa- 
tion vector y = $*r, or all of its nonzero coordinates, whichever set is smaller. 
Regularize: Among all subsets Jq C J with comparable coordinates: 



and the runtime is similar to that of OMP. There is, however, room for improvement which leads 
us to the CoSaMP algorithm, developed by Needell and Tropp [9]. CoSaMP provides optimal 
guarantees as well as an important implementation analysis. 



CoSaMP is an iterative recovery algorithm that provides the same guarantees as even the best 
optimization approaches. As in the case of ROMP and the Li approach, CoSaMP recovers signals 
using measurement matrices that satisfy the RIC. Thus as before, the observation vector y = 
serves as a good proxy for the signal x. Using the largest coordinates, an approximation to the 
signal is formed at each iteration. After each new residual is formed, reflecting the missing portion 
of the signal, the measurements are updated. This is repeated until all the recoverable portion of 
the signal is found. (See [9] for halting criteria.) The algorithm is described in Algorithm 4.2. 

For a recovery algorithm to be used efficiently in practice, the least squares step must be analyzed 
carefully. By the restricted isometry condition, the matrix in the estimation step is very well- 
conditioned. This suggests the use of an iterative method such as Richardson's iteration [1, Sec. 7.4] 
to apply the psuedoinverse = This method is analyzed in the context of CoSaMP 

and shown to provide an efficient means of acquiring the estimation [9]. The following theorem 
from [9] summarizes the fundamentally optimal recovery guarantees and rigorous computational 
costs of CoSaMP. 

Theorem 4.1 (Recovery by CoSaMP [9]). Suppose that $ is an m x N measurement matrix 
satisfying the restricted isometry condition with parameters (2s, c). Let u = + e be a vector of 
samples of an arbitrary signal, contaminated with arbitrary noise. For a given precision parameter 
r], the algorithm CoSaMP produces an s-sparse approximation a that satisfies 



The running time is 0(^ • log(||a;||2 /??)), where ^ bounds the cost of a matrix-vector multiply with 
$ or Working storage is 0{N). 

In [8] it is shown that only a fixed number of iterations is required to reduce the error to an 
optimal amount. The report [8] also discusses variations on the CoSaMP algorithm that may 
improve results in practice. 



|2/(«)| < 2|y(j)| for ah i,j G Jq, 

choose Jo with the maximal energy ||y|jo||2. 
Update: Add the set Jq to the index set: I <— lU Jq, and update the residual: 

w = arg min \\u — ^z\\2; r = u — ^w. 



4. CoSaMP 
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Algorithm 4.2: CoSaMP Recovery Algorithm [9] 



CoSaMP(*, u, s) 

Input: Sampling matrix noisy sample vector u, sparsity level s 
Output: An s-sparse approximation a of the target signal 


a'^ ^ 0, V ^ u, k ^ 


{ Trivial initial approximation } 


repeat 




k^ k + 1 




y ^ 


{ Form signal proxy } 


r2 ^ supp(y2s) 


{ Identify large components } 


T supp(a'=-^) 


{ Merge supports } 




{ Signal estimation by least-squares } 






a'' ^ bs 


{ Prune to obtain next approximation } 


V ^ u — <^a^ 


{ Update current samples } 


until halting criterion true 
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